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ABSTRACT 

The use of Post-Flight Feedback (PFFB) and 
Above Real-Time Training (ARTT) while 
training novice pilots to perform a 
coordinated level turn on a PC-based flight 
simulator was investigated. One group 
trained at 1.5 ARTT followed by an equal 
number of flights at 2.0 ARTT; the second 
group experienced Real Time Training 
(RTT). The total number of flights for both 
groups was equal. Each group was further 
subdivided into two groups one of which 
was provided PFFB while the other was not. 
Then, all participants experienced two 
challenging evaluation missions in real time. 
Performance was assessed by comparing 
root-mean-square error in bank-angle and 
altitude. Participants in the 1. 5/2.0 ARTT 
No-PFFB sequence did not show 
improvement in performance across training 
sessions. An ANOVA on performance in 
evaluation flights found that the PFFB 
groups performed significantly better than 
those with No-PFFB. Also, the RTT groups 
performed significantly better than the 
ARTT groups. Data from two additional 
groups trained under a 2.0/1 .5 ARTT PFFB 
and No-PFFB regimes were collected and 


combined with data from the previously 
studied groups and reanalyzed to study the 
influence of sequence. An ANOVA on test 
trials found no significant effects between 
groups. Under training situations involving 
ARTT we recommend that appropriate 
PFFB be provided. 

INTRODUCTION 

Advancements in computer technology have 
made the use of computer-based simulators 
and trainers more feasible for research 
investigating factors related to pilot training 
effectiveness. One of the training strategies 
utilized for skills acquisition is that of 
ARTT in which training skills are acquired 
on a real-time simulator when it is modified 
to present events faster than normal. The 
motivating factors for ARTT can be traced 
back to Kolf (1973) who noted that, 
"regardless of type or amount of pre-flight 
simulator training accomplished by the pilot, 
the actual flight appears to take place at a 
much faster time frame than real time" and 
Hoey (1976) who reported that the mental 
state of test pilots operating remotely piloted 
vehicles can be approximately simulated 
without stressful conditions by increasing 
the simulated rate of time passage. More 
recently, Crane and Guckenberger (1997) 
reported that pilots trained using ARTT 
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performed emergency procedures and 

defeated bandit aircraft significantly faster 
than pilots trained in real time. Rossi et al. 
(1999) trained university students on a 
gunneiy task to compare RTT and ARTT at 
1.5 times real time. The students trained in 
ARTT performed on test trials as well as 
students trained in RTT, although the ones 
trained in ARTT spent less clock time. 
During training, the performance of students 
in ARTT was depressed compared to those 
trained in RT. The authors suggested that 
using ARTT as top-off training after RTT 
might result in more effective training. 
Williams (1999) also observed that ARTT as 
top-off training after RTT offers better 
training in comparison to ARTT alone or 
RTT alone in a similar gunnery task. Ali, 
Guckenberger, Rossi, and Williams (2000) 
addressed the use of ARTT for training of 
pilots to perform basic flight maneuvers. 
They classified the flying maneuvers with 
reference to Fitts and Posner's (1967) model 
that recognizes different stages of skill 
acquisition as cognitive, associative, and 
autonomous. Ali et al. (2000) observed that 
using ARTT at the cognitive stage was 
beneficial for acquiring skills in straight- 
and-level flight. However, as climb and 
descent and level turns are relatively more 
complex than straight-and-level flying, they 
propose that ARTT is beneficial at the 
autonomous stage or as top-off training after 
RTT. 

Self-instruction through the use of feedback 
is increasingly being incorporated in 
computer-based learning. Proctor and Dutta 
(1995) provide a comprehensive discussion 
of the influence of feedback on motor skill 
acquisition. Ali et al. (2000) used two forms 
of feedback; first, automated in-flight aural 
cues that signaled to the pilot trending out of 
prescribed tolerances in altitude and heading 
and second, post-flight feedback in the form 
of strip-charts consisting of a graphical 
comparison of altitude, heading, airspeed, 
and other parameters during flight with the 
target values/tolerances. They observed that 
these two feedback strategies in general 
improved the performance. 


In summary, previous studies have 

determined that the efficacy of training 
depended on types of feedback, the type of 
piloting tasks, and the use of ARTT. The 
interaction of these parameters has not been 
systematically studied for training of novice 
pilots. 

Objectives 

The current investigation consisted of two 
related studies that attempted to: 

(a) understand the influence of different 
sequences of ARTT values, and 

(b) determine the adequacy of PFFB. 

EXPERIMENTAL SETUP 

Mock setup of a partial cockpit housed in 
the Flight Vehicle Lab at Tuskegee 
University was used as the pilot training 
station. This setup provides an out-the- 
window (OTW) panoramic view on three 
monitors with a heads-up display (HUD) on 
the center monitor. The heads-down display 
(HDD) of a conventional instrument panel 
was on a fourth monitor located below the 
OTW monitors. The four monitors are 
controlled by a Quantum 3D Heavy Meta! 
Computer having two Pentium II 400 MHz 
processors, 400 MB RAM, three extra 
display cards for Open GVS, based graphics, 
and a Sound Blaster audio card. The 
computer was configured by SDS 
International, Orlando, Florida and runs the 
Lite Flite version 3.3 (199) flight simulation 
software by SDS International. Lite Flite 
offers flight simulation of several aircraft 
including a Predator unmanned air vehicle 
(UAV) which was used in these studies. The 
controls include a Saitek X36F joystick, a 
Saitek X35T throttle, and CH rudder pedals. 

EXPERIMENT 1 

The purpose of this experiment was to 
investigate the effects of increasing values 
of ARTT in comparison to RTT, and the 
effects of PFFB vs. no PFFB during the 
training of a standard two-minute turn with 
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novice pilots. As previous studies have 
suggested ARTT to be useful for top-off 
training, the use of increasing values of 
ARTT may also prove beneficial. 

Participants 

Twenty undergraduate college students 
enrolled at Tuskegee University who had 
little or no prior flying experience served as 
novice pilots in exchange for course credit. 

Experimental Design 

This experiment consisted of a 2 X 2 X 2 
mixed factorial design. The between 
subjects variables were the type of training 
(RTT vs. ARTT) and feedback condition 
(PFFB vs. No-PFFB). The within subjects 
variable was the two evaluation flights. 

There were five participants in each group, 
with the exception of four in a RTT, no 
feedback (No-PFFB) group. 

Procedure 

All participants experienced the following 
segments: Orientation to Simulator Controls 
and Functions, Demonstration Flight, 
Familiarization Flights, Training Flights, 
and Evaluation Flights. 

Orientation to Simulator Controls and 
Functions. Demonstration and 
Familiarization. During the orientation 
process, participants were informed of the 
basic control surfaces of the aircraft, their 
functions, and the movements associated 
with each control surface. The participants 
then were given an overview of the locations 
and functions of the joystick, rudder pedals, 
and the throttle located in the mock cockpit. 
Then, participants were instructed in the 
location and functions of the following 
displays on the HUD: altimeter, radio 
altimeter, airspeed indicator, heading tape, 
artificial horizon, pitch ladder, and clock. 
They were also instructed on the location 
and functions of the following instruments 
on the HDD: artificial horizon and the 
vertical velocity indicator. Because the turn- 
and-slip indicator on the HDD was not 


functioning properly, the use of the HUD to 
determine if the aircraft was making a 
coordinated turn was explained. 

Participants observed the experimenter fly 
one 3-minute flight, while explaining the 
controls and operation of the aircraft. 
Participants flew four flights of 3-minute 
duration each in which they were told to 
maintain airspeed of 129 knots, a heading of 
360 degrees, and an altitude of 5,000 feet. 
Participants who achieved a score of 2.0 
(grade based on comparing parameter values 
in the flight with the parameter values and 
tolerances in an input file) on at least one 
familiarization flight continued into training. 
Twenty participants, who achieved the 
passing grade continued on to the training 
phase. 

Training and Evaluation . The experiment 
consisted of a training phase and an 
evaluation phase. The training mission 
consisted of a coordinated 1 80° turn with a 
1 0° bank angle while the evaluation task was 
an S-turn with a bank angle of 30°. 

However, a loss of situational awareness 
was observed in most of the participants 
after the first leg of the S-turn. Thus, only 
the first leg of the S-turn was evaluated for 
analysis purposes. Each group conducted 
their training in the following sequence 
immediately followed by two evaluation 
flights in real time. 

Group I: Five missions in 1 .5 ARTT then 
five missions in 2.0 ARTT with No-PFFB 

Group II: Five missions in 1 .5 ARTT then 
five missions in 2.0 ARTT with PFFB 

Group III: Five missions in RTT then five 
missions in RTT with No-PFFB 

Group IV: Five missions in RTT then five 
missions in RTT with PFFB 

Feedback. Participants in the No-PFFB 
groups received no feedback during or after 
flights. If they asked questions about the 
task, they were reread the relevant 
instructions. Participants in the PFFB groups 
received verbal feedback after each flight 
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consisting of information about errors in 

manipulating the controls and in focusing on 
the wrong instruments. After the second, 
fourth, sixth, eighth, and tenth flights, 
participants in the PFFB groups were also 
shown a printout of the desired ground track 
and the pilot's actual ground track for 
comparison. Then, the instructor explained 
possible reasons for the deviations in 
performance 

Performance Measure . Performance of the 
pilots was assessed by comparing a ‘root- 
mean-square’ error in the bank-angle, and 
altitude (normalized by the radius of turn) 
calculated every three seconds of flight and 
averaged over the duration of the flight. In 
general the error in velocity should also 
have been included but it was observed that 
once the pilot initiated a turn, there was no 
attempt to manipulate the throttle, as a 
consequence of which the changes in 
altitude and speed were correlated (Figure 
1 ). 

More details of the selection, training, and 
evaluation process are given in Ali et al. 
(2003). 



Figure 1: Varaitlon of Altitude & Speed 


Results and Discussion 

A three-way , split plot ANOVA was 
conducted with one within-subjects factor, 
evaluation flight #1 vs. evaluation flight #2, 
and two between-subjects factors, training 
time and feedback condition. There was no 
significant difference in performance scores 
between the two evaluation flights, F(l, 1 5) 
< 1, and interactions between evaluation 
trials and training time, feedback condition, 
and evaluation time by feedback condition 
interaction were all not significant (all F 
values < 1). For the between-subjects 
factors there was no significant interaction 
between training time and feedback 
condition, F (1 , 15) = 2.99. However, there 
was a significant difference in performance 
scores between feedback and No-PFFB 
groups, F (1 , 1 5) = 6.4 1 8, p < .05. 
Specifically, participants in the PFFB groups 
performed better than those in the No-PFFB 
groups in the evaluation flights. In addition, 
there was a significant difference in 
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performance scores for training time, i.e., 
participants in the RTT/RTT groups 
performed better than those in the ARTT 
groups on evaluation trials, F(1 ,1 5) = 7.009, 
p < .05. 

The various training interventions were 
observed to influence performance across 
training flights and the evaluation flights 
(Figure 2). The participants who 
experienced RTT without feedback may be 
viewed as a control group and it may be 
seen that their performance improved with 
practice in the absence of extrinsic feedback; 
however, this improvement did not appear 
substantially until the eighth training 
session. In general, participants in the 
feedback groups improved performance 
early in training, and performed better than 
their counterparts across training sessions. 
However, as can be observed from Figure 2, 
the error in the evaluation flights for both 
the RTT groups was higher than the errors 
during training. This is attributed to the 
increased complexity of the task. The group 


which received ARTT without feedback 
( 1 .5/2.0 No-PFFB) did not exhibit 
improvement with time, and worsened in 
performance after switch over from 1 .5 to 
2.0 ARTT. This perhaps was due to the 
increased difficulty of the task due to the 
switchover to a higher value of ARTT 
conditions. 

It was also observed that the 1 .5/2.0 No- 
PFFB group performed the worst in the 
evaluation flights (Figure 2), exhibiting a 
poorer transfer of training from training 
conditions to evaluation conditions. Perhaps 
responses acquired during the ARTT carried 
over into the evaluation flights in RT. 

EXPERIMENT 2 

In the previous study participants who 
experienced ARTT values of 1.5 followed 
by 2.0 were compared to those who 
experienced real-time training. The purpose 
of this second study was to expand the 
previous study by comparing the previous 



Figure 2 : Average Errors During Training & Evaluation Flights 
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four groups with two additional groups of 
participants who received ARTT at a value 
of 2.0 followed by 1 .5. One group received 
feedback and the other did not. 

Participants, Experimental Design and 

Procedure 

Two groups of five novice pilots in each 
participated, and their data was combined 
with the previous four groups, making a 
total of 30 participants. Thus, with the 
addition of the two new groups, the design 
was a 3 X 2 X 2 mixed factorial design, with 
the ARTT condition ( 1 .5/2.0, 2.0/1 .5, 

1 .0/1 .0) and feedback (PFFB/No-PFFB) as 
the between subjects factors and evaluation 
trial as the within subjects factor. The 
procedure was exactly the same as in the 
previous study. 

Results and Discussion 

Participants in the 2.0/1. 5 ARTT sequence 
exhibited some interesting characteristics. 
The No-PFFB 2.0/1. 5 ARTT group 
demonstrated a continuous improvement in 
performance over the training flights even 
after the switch from 2.0 to 1.5 ARTT. 
However, in comparison to the switch over 
error during training, there was a relatively 
larger error exhibited during the evaluation 
flights which occurred in RT. This again 
suggests that performance in the evaluation 
flights was being affected by both transfer of 
training effects as well as a change in task 
complexity. It is however noted that this 
impact is lesser as compared to the larger 
change of slowing down from 2.0 ART to 
RT as was in the case of the 1 .5/2.0 group. 
The performance by the 2. 0/1. 5 ARTT with 
PFFB group showed the impact of PFFB 
through continued improvement as training 
progressed which is consistent with the other 
PFFB groups. However, its performance 
during the evaluation flights was observed to 
be not as good as the 2.0/1 .5 No-PFFB 
group. This is attributed to the fact that the 
PFFB group was not as strong (the PFFB 
group had an initial average error of 0.25 as 
compared to 0.14 for the No-PFFB group). 
Thus, performance on the evaluation task 


(which was more challenging) was more 
depressed. 

A split-plot ANOVA was carried out 
with evaluation trial as the within subjects 
factor and ARTT condition and feedback 
condition as the between subjects factors. 
There was no significant difference between 
evaluation trials and the interactions 
between evaluation trials and the other 
factors were not significant. The between 
subjects effects were also not significant. An 
analysis of the means of the evaluation 
trials, however, shows that the 1 .5/2.0 No- 
PFFB group performed considerably worse 
than all other groups. 

SUMMARY AND CONCLUSIONS 

In summary, the analysis suggests that for a 
coordinated level turn: 

1 . 1 .5/2.0 ARTT with No-PFFB is not an 
effective training strategy; 

2. ARTT does not seem to offer any 

advantage over RTT for this maneuver; 

3. ARTT should be used in conjunction 

with appropriate PFFB; 

4. PFFB in RTT is the most efficient 

combination. 

Future work should address a) whether the 
depressed performance of ARTT groups 
continues with further evaluation trials, b) if 
pilots trained with one value of ARTT 
throughout training perform better than 
those with RT, and c) whether ARTT is 
beneficial for other flying maneuvers. 
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